We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy are not extendable to infinite state spaces, as the noise level to ensure privacy will scale accordingly to infinity. Our aim is to protect the value function approximator, without regard to the number of states queried to the function. It is achieved by adding functional noise to the value function iteratively in the training. We show rigorous privacy guarantees by a series of analyses on the kernel of the noise space, the probabilistic bound of such noise samples, and the composition over the iterations. We gain insight into the utility analysis by proving the algorithm's approximate optimality when the state space is discrete. Experiments corroborate our theoretical findings and show improvement over existing approaches.

The definition of two neighboring reward functions is provided in Theorem 5. The authors did not explain the motivation of the guarantee of privacy for reward function clearly. It would be better if the authors could interpret the necessities of the privacy of reward function in some real application situations. What is the reason for adding noise like line 19-20 of the Algorithm 1? The definitions of g _k[B][2] in line 4 and g _a[:][1] in line 15 are not given.

This paper proposes a differentially private Q-learning algorithm for RL with continuous observations. This is a nice application of the functional Gaussian noise mechanism, and the paper provides a rigorous privacy and utility analysis. When preparing the final version the authors should fix the presentation issues raised in the reviews, and make sure the paper is properly positioned wrt previous work (eg. BGP'16 used a stricter notion of neighbouring relation between datasets that includes changes in the states and actions, not only rewards).

We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy are not extendable to infinite state spaces, as the noise level to ensure privacy will scale accordingly to infinity. Our aim is to protect the value function approximator, without regard to the number of states queried to the function. It is achieved by adding functional noise to the value function iteratively in the training.

We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy are not extendable to infinite state spaces, as the noise level to ensure privacy will scale accordingly to infinity. Our aim is to protect the value function approximator, without regard to the number of states queried to the function. It is achieved by adding functional noise to the value function iteratively in the training. We show rigorous privacy guarantees by a series of analyses on the kernel of the noise space, the probabilistic bound of such noise samples, and the composition over the iterations.